Remote Direct Memory Access (RDMA) is a protocol which may be run over TCP/IP to provide memory copy and message passing semantics. The RDMA specification may be found at http://www.rdmaconsortium.org. With inherent memory copy semantics, remote direct memory access provides the ability of having dedicated hardware to directly place data into an application buffer without host CPU involvement. Each connection may comprise a queue pair (QP) in RDMA terminology and each QP consists of a send queue (SQ), a receive queue (RQ) and one completion queue (CQ). RDMA aware applications may initiate requests to either the SQ or the RQ, and harvest the results of the requests from within the CQ when the RDMA aware network interface card (RNIC) posts a completion record to CQ. Each entry in the SQ and RQ may be referred to as a work request entry, or WQE, whereas each entry in the CQ may be referred to as a completion queue entry, or CQE. Each of the queue pairs (QPs) may be in one of a plurality of states. RDMA memory copy and message passing semantics may only be carried out in the RTS (Ready To Send) state. The full RDMA state is documented in the RDMA Verb Specification, which is incorporated herein by reference in its entirety.
FIG. 1 is a block diagram of a conventional RDMA QP state machine 100. An RNIC interface (RI) may restrict the QP to be in only one of five states. Referring to FIG. 1, the RDMA QP state machine 100 may process a given QP, which may be in an idle state 101, a RTS state 104, a terminating state 102, an error state 103, and/or a closing state 105.
A QP may be created by a Create QP command 107. At the time of creation, the QP is in the idle state 101. A QP may also transition from another state to the idle state 101 after application of a Modify QP verb. For example, the Modify QP verb 109 may transition the QP state from idle state 101 to idle state 101 in order to allow QP context attributes to be modified while in idle state. The Modify QP verb 111 may transition the QP state from the idle state 101 to the RTS state in order to allow support for low lever protocol (LLP) establishment. The Modify QP verbs 113 and 115 may be utilized if WQEs posted to a QP while in idle state have to be removed from the QP. A Modify QP verb 113 may be utilized to transition the QP state from the idle state 101 to the error state 103 and then back to the idle state 101 via the Modify QP verb 115.
The RTS state 104 may provide normal message processing for all SQ and RQ messages. Several verb commands may be applied to a QP while in the RTS state 104. For example, a Modify QP verb 117 may be utilized to transition a QP from the RTS state 104 to the RTS state 104. Such transition may allow for certain QP parameters to be changed while a QP is in associated with another QP. A Modify verb 119 may be utilized to transition the QP state from the RTS state 104 to the closing state 105 whenever the SQ work requests and RDMA read operations are completed. If the SQ work requests or the RDMA read operations are not completed, or if a terminate message has been received via a QP, the Modify QP verb 121 may be utilized to transition the QP from the RTS state 104 to the terminate state 102. A transition to the terminate state 102 may also allow a consumer to notify a remote peer of an abnormal termination of a connected stream. A Modify verb 123 may be utilized to transition the QP from the RTS state 104 to the error state 103 in order to allow upper layer protocol (ULP) abortive teardown.
The terminate state 102 may be utilized to send final terminate messages and to begin an LLP Close processing if a terminate message is received from a remote peer. The terminate state 102 may also be exited to the error state 103. If the terminate state 102 is exited to the error state 103, the LLP stream may no longer be associated with the QP and the LLP stream may be in either a condition of LLP Closed or LLP Reset, 125.
The error state 103 may provide an indication that a QP has experienced an error and has stopped operation. Upon entry of the error state 103, the LLP stream may not be associated with the QP. While in the error state 103, a Modify QP verb 131 may be utilized to transition the QP from the error state 103 to the error state 103.
The closing state 105 may be utilized to wait for an LLP to complete an LLP Close cycle, if there are no errors. When the closing state 105 is exited to the idle state 101 via the transition 127, the LLP stream may not be associated with the QP. Errors detected by an RI whenever a QP is in the closing state 105 may result in a transition 129 to the error state 103.
A conventional RDMA QP state machine may be implemented in a single RNIC so that the RNIC may process a QP in an idle state, a RTS state, a terminating state, an error state, and/or a closing state. However, RDMA QP state processing on the RNIC may require utilization of significant resources and/or hardware on the RNIC.
Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of such systems with some aspects of the present invention as set forth in the remainder of the present application with reference to the drawings.